Apparatus for eye tracking

ABSTRACT

An eye tracker having a waveguide for propagating illumination light towards an eye and propagating image light reflected from at least one surface of an eye, a light source optically coupled to the waveguide, and a detector optically coupled to the waveguide. Disposed in the waveguide is at least one grating lamina for deflecting the illumination light towards the eye along a first waveguide path and deflecting the image light towards the detector along a second waveguide path.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority from U.S. Provisional Application Ser.Nos. 61/688,300 with filing date 11 May 2012 entitled APPARATUS FOR EYETRACKING, which is hereby incorporated by reference in its entirety.

The following patent applications are incorporated by reference hereinin their entireties: U.S. Provisional Patent Application No. 61/687,436with filing date 25 April 12 entitled WIDE ANGLE COLOUR HEAD MOUNTEDDISPLAY.

U.S. Provisional Patent Application No. 61/689,907 with filing date 25April 12 entitled HOLOGRAPHIC HEAD MOUNTED DISLAY WITH IMPROVED IMAGEUNIFORMITY.

U.S. Provisional Application No. 61/796,632 with filing date 16 Oct.2012 entitled TRANSPARENT DISPLAYS BASED ON HOLOGRAPHIC SUBSTRATE GUIDEDOPTICS.

U.S. Provisional Application No. 61/849,853 with filing date 4 Feb. 2013entitled TRANSPARENT WAVEGUIDE DISPLAY.

PCT Application No.: US2008/001909, with International Filing Date: 22Jul. 2008, entitled LASER ILLUMINATION DEVICE.

PCT Application No.: US2006/043938, entitled METHOD AND APPARATUS FORPROVIDING A TRANSPARENT DISPLAY.

PCT Application No.: PCT/GB2010/001982 entitled COMPACT EDGE ILLUMINATEDEYEGLASS DISPLAY.

PCT Application No.: PCT/GB2012/000680, entitled IMPROVEMENTS TOHOLOGRAPHIC POLYMER DISPERSED LIQUID CRYSTAL MATERIALS AND DEVICES.

PCT Application No.: PCT/GB2010/000835 entitled COMPACT HOLOGRAPHIC EDGEILLUMINATED EYEGLASS DISPLAY.

U.S. Pat. No. 6,115,152 entitled HOLOGRAPHIC ILLUMINATION SYSTEM.

BACKGROUND OF THE INVENTION

This invention relates to an eye tracking sensor, and more particularlyto an eye tracker using electrically switchable gratings.

Eye tracking is important in Head Mounted Displays (HMDs) because it canextend the pilot's ability to designate targets well beyond the headmobility limits. Eye tracking technology based on projecting IR lightinto the users eye and utilizing the primary Purkinje (corneal)reflection and the pupil-masked retina reflection have been around sincethe 1980's. The method tracks the relative motion of these images inorder to establish a vector characterizing the point of regard. Most eyetrackers have employed flat beam splitters in front of the users' eyesand relatively large optics to image the reflections onto a sensor(generally a CCD or CMOS camera).

There is much prior art in the patent and scientific literatureincluding the following United States filings:

-   -   1. United Stated Patent Application Publication No.        US2011019874(A1) by Levola et al entitled DEVICE AND METHOD FOR        DETERMINING GAZE DIRECTION;    -   2. U.S. Pat. No. 5,410,376 by Comsweet entitled EYE TRACKING        METHOD AND APPARATUS;    -   3. U.S. Pat. No. 3,804,496 by Crane et al entitled TWO        DIMENSIONAL EYE TRACKER AND METHOD FOR TRACKING AN EYE TWO        DIMENSIONAL EYE TRACKER AND METHOD FOR TRACKING AN EYE;    -   4. U.S. Pat. No. 4,852,988 by Velez et al entitled VISOR AND        CAMERA PROVIDING A PARALLAX-FREE FIELD-OF-VIEW IMAGE FOR A        HEAD-MOUNTED EYE MOVEMENT MEASUREMENT SYSTEM;    -   5. U.S. Pat. No. 7,542,210 by Chirieleison entitled EYE TRACKING        HEAD MOUNTED DISPLAY;    -   6. United Stated Patent Application Publication No. US        2002/0167462 A1 by Lewis entitled PERSONAL DISPLAY WITH VISION        TRACKING; and    -   7. U.S. Pat. No. 4,028,725 by Lewis entitled HIGH RESOLUTION        VISION SYSTEM.

The exit pupil of these trackers is generally limited by either the sizeof the beamsplitter or the first lens of the imaging optics. In order tomaximize the exit pupil, the imaging optics are positioned close to thebeamsplitter, and represent a vision obscuration and a safety hazard.Another known limitation with eye trackers is the field of view, whichis generally limited by the illumination scheme in combination with thegeometry of the reflected images off the cornea. The cornea is anaspheric shape of smaller radius that the eye-ball. The corneareflection tracks fairly well with angular motion until the reflectedimage falls off the edge of the cornea and onto the sclera. The need forbeam splitters and refractive lenses in conventional eye trackersresults in a bulky component that is difficult to integrate into a(HMD). The present invention addresses the need for a slim, wide fieldof view, large exit pupil, high-transparency eye tracker for HMDs.

The inventors have found the diffractive optical elements offer a routeto providing compact, transparent, wide field of view eye trackers. Onimportant class of diffractive optical elements is based on SwitchableBragg Gratings (SBGs). SBGs are fabricated by first placing a thin filmof a mixture of photopolymerizable monomers and liquid crystal materialbetween parallel glass plates. One or both glass plates supportelectrodes, typically transparent indium tin oxide films, for applyingan electric field across the film. A volume phase grating is thenrecorded by illuminating the liquid material (often referred to as thesyrup) with two mutually coherent laser beams, which interfere to form aslanted fringe grating structure. During the recording process, themonomers polymerize and the mixture undergoes a phase separation,creating regions densely populated by liquid crystal micro-droplets,interspersed with regions of clear polymer. The alternating liquidcrystal-rich and liquid crystal-depleted regions form the fringe planesof the grating. The resulting volume phase grating can exhibit very highdiffraction efficiency, which may be controlled by the magnitude of theelectric field applied across the film. When an electric field isapplied to the grating via transparent electrodes, the naturalorientation of the LC droplets is changed causing the refractive indexmodulation of the fringes to reduce and the hologram diffractionefficiency to drop to very low levels. Note that the diffractionefficiency of the device can be adjusted, by means of the appliedvoltage, over a continuous range. The device exhibits near 100%efficiency with no voltage applied and essentially zero efficiency witha sufficiently high voltage applied. In certain types of HPDLC devicesmagnetic fields may be used to control the LC orientation. In certaintypes of HPDLC phase separation of the LC material from the polymer maybe accomplished to such a degree that no discernible droplet structureresults.

SBGs may be used to provide transmission or reflection gratings for freespace applications. SBGs may be implemented as waveguide devices inwhich the HPDLC forms either the waveguide core or an evanescentlycoupled layer in proximity to the waveguide. In one particularconfiguration to be referred to here as Substrate Guided Optics (SGO)the parallel glass plates used to form the HPDLC cell provide a totalinternal reflection (TIR) light guiding structure. Light is “coupled”out of the SBG when the switchable grating diffracts the light at anangle beyond the TIR condition. SGOs are currently of interest in arange of display and sensor applications. Although much of the earlierwork on HPDLC has been directed at reflection holograms transmissiondevices are proving to be much more versatile as optical system buildingblocks.

Typically, the HPDLC used in SBGs comprise liquid crystal (LC),monomers, photoinitiator dyes, and coinitiators. The mixture frequentlyincludes a surfactant. The patent and scientific literature containsmany examples of material systems and processes that may be used tofabricate SBGs. Two fundamental patents are: U.S. Pat. No. 5,942,157 bySutherland, and U.S. Pat. No. 5,751,452 by Tanaka et al. both filingsdescribe monomer and liquid crystal material combinations suitable forfabricating SBG devices.

One of the known attributes of transmission SBGs is that the LCmolecules tend to align normal to the grating fringe planes. The effectof the LC molecule alignment is that transmission SBGs efficientlydiffract P polarized tight (ie light with the polarization vector in theplane of incidence) but have nearly zero diffraction efficiency for Spolarized tight (ie light with the polarization vector normal to theplane of incidence. Transmission SBGs may not be used at near-grazingincidence as the diffraction efficiency of any grating for Ppolarization falls to zero when the included angle between the incidentand reflected light is small.

There is a requirement for a compact, lightweight eye tracker with alarge field of view, and a high degree of transparency to externallight.

SUMMARY OF THE INVENTION

It is a first object of the invention to provide a compact, lightweighteye tracker with a large field of view, and a high degree oftransparency to external light.

It is a second object of the invention to provide a compact, lightweighteye tracker with a large field of view, and a high degree oftransparency to external light implemented in a thin optical waveguide.

The objects of the invention are achieved in one embodiment of theinvention in which there is provided a eye tracker comprising: awaveguide for propagating illumination light towards an eye andpropagating image light reflected from at least one surface of an eye; alight source optically coupled to the waveguide; a detector opticallycoupled to the waveguide. Disposed in the waveguide is at least onegrating lamina for deflecting the illumination light towards the eyealong a first waveguide path and deflecting the image light towards thedetector along a second waveguide path.

In one embodiment of the invention the first and second waveguide pathsare in opposing directions.

In one embodiment of the invention at least one portion of the at leastone grating lamina deflects the illumination light out of the firstwaveguide path and at least one portion of the at least one gratinglamina deflects the image light into the second waveguide path.

In one embodiment of the invention the grating lamina comprises amultiplicity of electrically switchable elements each having adiffracting state and a non diffracting state. The at least one portionof the at least one grating lamina is a grating element in itsdiffracting state.

In one embodiment of the invention the electrically switchable elementsare elongate with longer dimension aligned perpendicular to at least oneof the first and second waveguide paths.

In one embodiment of the invention the at least one grating laminacomprises an illumination grating for deflecting the illumination lightin the first waveguide path towards the eye and an imaging grating fordeflecting the image light into the second waveguide path.

In one embodiment of the invention at least one grating lamina furthercomprises at least one of an input grating for deflecting illuminationlight from the source into the first waveguide path and an outputgrating for deflecting the image light out of the second waveguide pathtowards the detector.

In one embodiment of the invention the eye tracker further comprises animage sampling grating overlaying the output grating. The image samplinggrating comprises a linear array of switchable grating elements. Eachgrating element when in its diffracting state samples a portion of thelight in the waveguide and deflects it along the image sampling gratingtowards the detector.

In one embodiment of the invention the eye tracker further comprises anillumination sampling grating overlaying the input grating. Theillumination sampling grating is optically coupled to the light source.The illumination sampling grating comprises a linear array of switchablegrating elements. Each grating element when in its diffracting statedeflects light from the illumination sampling grating into thewaveguide.

In one embodiment of the invention the illumination grating abuts anupper or lower edge of the imaging grating along the first waveguidepath.

In one embodiment of the invention the illumination grating comprisesfirst and second gratings disposed adjacent upper and lower edges of theimaging grating along the first waveguide path.

In one embodiment of the invention the imaging grating comprises a firstarray of switchable elongate beam deflection grating elements and anoverlapping second array of switchable elongate beam deflection gratingelements. The elements of the first and second arrays are disposed withtheir longer dimensions orthogonal.

In one embodiment of the invention the illumination grating is a lineararray of elongate switchable beam deflection elements with longerdimension aligned perpendicular to the first waveguide path.

In one embodiment of the invention the at least one grating lamina isone of a switchable Bragg grating, a switchable grating recorded in areverse mode holographic polymer dispersed liquid crystal, or a nonswitching Bragg grating.

In one embodiment of the invention the image light is speckle.

In one embodiment of the invention the eye surface providing the imagelight is at least one of the cornea, lens, iris, sclera and retina.

In one embodiment of the invention the detector is a two dimensionalarray.

In one embodiment of the invention the at least one grating laminaencodes optical power.

In one embodiment of the invention the detector is connected to an imageprocessing apparatus for determining at least one spatio-temporalcharacteristic of an eye movement.

In one embodiment of the invention the image light is a Purkinjereflection.

In one embodiment of the invention the source is a laser.

In one embodiment of the invention source is a light emitting diode.

In one embodiment of the invention the illumination grating providescollimated light.

In one embodiment of the invention the illumination grating providesdivergent light.

In one embodiment of the invention the imaging grating encodes opticalpower.

In one embodiment of the invention the illumination grating encodesoptical power.

In one embodiment of the invention the illumination, imaging, input andoutput gratings are co planar.

In one embodiment of the invention the input and illumination gratingslie in a first plane and the imaging and output gratings lie in a secondplane parallel to the first plane.

A more complete understanding of the invention can be obtained byconsidering the following detailed description in conjunction with theaccompanying drawings, wherein like index numerals indicate like parts.For purposes of clarity, details relating to technical material that isknown in the technical fields related to the invention have not beendescribed in detail.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a schematic plan view of an eye tracker shown in relation toa human eye in one embodiment of the invention

FIG. 1B is a schematic front elevation view showing elongate gratingelements used in the imaging grating in one embodiment of the invention.

FIG. 1C is a schematic front elevation view showing a two dimensionalarray of grating elements used in the imaging grating in one embodimentof the invention.

FIG. 2 is a schematic plan view of the eye tracker shown the imaging andillumination gratings and input and output gratings in one embodiment ofthe invention.

FIG. 3 is a schematic plan view of the eye tracker shown the imaging andillumination gratings and input and output gratings in one embodiment ofthe invention.

FIG. 4 is a plan view of the eye tracker shown the imaging andillumination gratings and input and output gratings in one embodiment ofthe invention,

FIG. 5 is a schematic cross section view of an illumination grating usedin one embodiment of the invention.

FIG. 6 is a schematic cross section view of an illumination grating usedin one embodiment of the invention.

FIG. 7 is a schematic cross section view of a first aspect of an imaginggrating used in one embodiment of the invention.

FIG. 8 is a schematic cross section view of a second aspect of animaging grating used in one embodiment of the invention.

FIG. 9A is a schematic top elevation view of a first layer of a twolayer imaging grating in one embodiment of the invention.

FIG. 9B is a schematic plan view of a first layer of a two layer imaginggrating in one embodiment of the invention.

FIG. 9C is a schematic side elevation view of a first layer of a twolayer imaging grating in one embodiment of the invention.

FIG. 10A is a schematic top elevation view of a second layer of a twolayer imaging grating in one embodiment of the invention.

FIG. 10B is a schematic plan view of a second layer of a two layerimaging grating in one embodiment of the invention.

FIG. 10C is a schematic side elevation view of a second layer of a twolayer imaging grating in one embodiment of the invention.

FIG. 11A is a schematic cross view of the human eye illustrating theformation of the Purkinje images.

FIG. 11B is a schematic cross view of the human eye illustratingreflections from the retina and iris.

FIG. 12A is a schematic plan view illustrating a first aspect of thelocalization of an eye feature by a two layer imaging grating each layercomprising elongate elements with the elements of the two gratings atright angle.

FIG. 12B is a schematic plan view illustrating a second aspect of thelocalization of an eye feature by a two layer imaging grating each layercomprising elongate elements with the elements of the two gratings atright angle.

FIG. 13A is a schematic cross of the human eye in a first rotationalstate showing a typical speckle pattern formed by the cornea and retina.

FIG. 13B is a schematic cross of the human eye in a first rotationalstate showing a typical speckle pattern formed by the cornea and retina.

FIG. 14 is a schematic front elevation view of a human eye show showingthe directions of motions of speckle patterns produced by the retina andcornea.

FIG. 15A is a schematic cross section view of a human eye in a firstrotational state showing reflection from the retina and cornea.

FIG. 15B is a schematic cross section view of a human eye in a secondrotational state showing reflection from the retina and cornea.

FIG. 16 is a schematic cross section view of an imaging gratingcomprising an array of SBG lens elements with focal length varyingacross the exit pupil in one embodiment of the invention.

FIG. 17A is a schematic cross section view of an imaging gratingcomprising an array of variable power lenses in one embodiment of theinvention.

FIG. 17B is a detail of FIG. 17A showing a variable power lenscomprising a variable index layer and a diffractive element of fixedfocal length.

FIG. 18 is a schematic illustrate of an imaging grating in oneembodiment of the invention in which the imaging grating comprises anarray of interspersed grating elements having at least two differentprescriptions.

FIG. 19 is a schematic cross section view of the illumination grating ofan eye tracker using separate illumination and imaging gratings in oneembodiment of the invention.

FIG. 20 is a schematic plan view the illumination grating of an eyetracker using separate illumination and imaging gratings in oneembodiment of the invention.

FIG. 21 is a schematic cross section view of an alternative illuminationgrating of an eye tracker using separate illumination and imaginggratings in one embodiment of the invention.

FIG. 22A is a schematic plan view of the imaging grating, the imagesampling grating and the detector module of an eye tracker usingseparate illumination and imaging gratings in one embodiment of theinvention.

FIG. 22B is a schematic plan view of image sampling grating and thedetector module of an eye tracker using separate illumination andimaging gratings in one embodiment of the invention.

FIG. 22C is a schematic cross section view of the imaging grating andthe image sampling grating of an eye tracker using separate illuminationand imaging gratings in one embodiment of the invention.

FIG. 22D is a schematic cross section view of image sampling grating andthe detector module of an eye tracker using separate illumination andimaging gratings in one embodiment of the invention.

FIG. 22E is a schematic cross section view of the imaging grating, theimage sampling grating and the detector module of an eye tracker usingseparate illumination and imaging gratings in one embodiment of theinvention.

FIG. 23 is a block diagram showing the principal modules of an eyetracker system using separate illumination and imaging gratings in oneembodiment of the invention.

FIG. 24 is a schematic illustration of a grating element switchingscheme provided by the imaging grating and image sampling grating in oneembodiment of the invention.

FIG. 25 is a schematic plan view of an eye tracker using commonillumination and imaging gratings in one embodiment of the invention.

FIG. 26 is a schematic cross section view showing the imaging andillumination grating and the input, output, image sampling and detectorsampling gratings of an eye tracker using common illumination andimaging gratings in one embodiment of the invention.

FIG. 27A is a schematic plan view of the image sampling grating of aneye tracker using common illumination and imaging gratings in oneembodiment of the invention.

FIG. 27B is a schematic cross section view of the illumination samplinggrating, the input grating and laser of an eye tracker using commonillumination and imaging gratings in one embodiment of the invention.

FIG. 27C is a schematic plan view image sampling grating and thedetector module with detector overlaid of an eye tracker using commonillumination and imaging gratings in one embodiment of the invention.

FIG. 27D is a schematic plan side elevation view showing the imagesampling grating and detector of an eye tracker using commonillumination and imaging gratings in one embodiment of the invention.

FIG. 27E is a schematic cross section view of the output grating and theimage sampling grating an eye tracker using common illumination andimaging gratings in one embodiment of the invention,

FIG. 27F is a schematic cross section view of the input grating and theillumination sampling of an eye tracker using common illumination andimaging gratings in one embodiment of the invention.

FIG. 28 is a simplified representation of the imaging process an eyetracker using common illumination and imaging gratings in one embodimentof the invention.

FIG. 29 provides a system block diagram showing the key modules of aneye tracker using common illumination and imaging gratings in oneembodiment of the invention.

FIG. 30 is a flow chart showing the process for determining eyedisplacement vectors from the recorded speckle data.

FIG. 31A is a flowchart for a calibration process for an eye trackerusing common illumination and imaging gratings in one embodiment of theinvention.

FIG. 31B is a schematic illustration of the initial calibrationprocedure used for an eye tracker in one embodiment of the invention

DETAILED DESCRIPTION OF THE INVENTION

The invention will now be further described by way of example only withreference to the accompanying drawings. It will apparent to thoseskilled in the art that the present invention may be practiced with someor all of the present invention as disclosed in the followingdescription. For the purposes of explaining the invention well-knownfeatures of optical technology known to those skilled in the art ofoptical design and visual displays have been omitted or simplified inorder not to obscure the basic principles of the invention. Unlessotherwise stated the term “on-axis” in relation to a ray or a beamdirection refers to propagation parallel to an axis normal to thesurfaces of the optical components described in relation to theinvention. In the following description the terms light, ray, beam anddirection may be used interchangeably and in association with each otherto indicate the direction of propagation of light energy alongrectilinear trajectories. Parts of the following description will bepresented using terminology commonly employed by those skilled in theart of optical design. It should also be noted that in the followingdescription of the invention repeated usage of the phrase “in oneembodiment” does not necessarily refer to the same embodiment.

The proposed eye tracker aims to satisfy a suite of challengingrequirements. Since it will eventually be integrated into a head-worncolor display, it should make minimum impact on the overall opticalperformance. The inventors' design goals are: a field of view (FOV) of60° horizontal×48° vertical; 17 mm eye relief; and eye motion box/exitpupil (20 mm.×10-15 mm). Moreover, the eye tracker must satisfy eyesafety requirements for near-eye visual displays with regard to weight(minimal), center of gravity (ergonomic), and profile. Furthermore itshould not compromise: pixel resolution, see-through (≧90%) and powerconsumption (minimal).

Eye Trackers based on classical Purkinje imaging methods suffer fromhigh latency resulting mainly from the large delay incurred by featurerecognition and tracking algorithms. The inventors are stronglymotivated by a desire to develop an eye tracker that firstly simplifiesthe image processing problems of classical eye tracking that oftenresult in unacceptably high latency and secondly can make use ofrelatively unsophisticated detector technology. Ideally the detectortechnology would be equivalent in specification to that used in theinfrared mouse a device which is now ubiquitous and more importantlycapable of being manufactured using sub dollar components. Although thepresent invention may be used to track eye movements using any type ofreflection from any surfaces of the eye (including reflections frommultiple surfaces and scatter from the optical media inside the eye) theinventors believe that tracking laser speckle reflected from the cornea,retina and other surfaces may offer significant. The inventors believethat detecting and processing speckle images is more efficient thanconventional video based technology in terms of detector resolution,processing overhead and power consumption.

An eye tracker according to the principles of the invention provides aninfrared illumination channel for delivering infrared illumination tothe eye and an imaging channel for forming an image of the eye at asensor. In one embodiment of the invention illustrated in FIGS. 1-2, theeye tracker comprises a waveguide 100 for propagating illumination lighttowards an eye 116 and propagating image light reflected from at leastone surface of an eye; a light source 112 optically coupled to thewaveguide; and a detector 113 optically coupled to the waveguide.Disposed in the waveguide are: at least one input grating 114 fordeflecting illumination light from the source into a first waveguidepath; at least one illumination grating 102 for deflecting theillumination light towards the eye; at least one imaging grating 101 fordeflecting the image light into a second waveguide path; and at leastone output grating 115 for deflecting the image light towards thedetector. The inventors also refer to the waveguide 100 as the DigiLens.The illumination and imaging gratings are arrays of switchable beamdeflection grating elements with the preferred grating technology beinga SBG as described above. In one embodiment of the invention shown inFIG. 1B the grating elements in the imaging grating 120 are elongate asindicated by 121 with longer dimension orthogonal to the beampropagation direction. In one embodiment of the invention shown in FIG.1C the imaging grating may comprise a two dimensional array 122 ofelements 123 each having optical power in two orthogonal planes.Typically the first and second waveguide paths, that is, the imaging andillumination paths in the waveguide are in opposing directions asillustrated in FIG. 1A. The illumination light will typically be fullycollimated while the image light will have some divergence of angledetermined by the scattering angle from eye services, the angularbandwidth of the gratings and the numerical aperture of the gratingelements. As will be discussed later, in one embodiment the imaging andillumination gratings are provided by a single grating with theillumination and imaging ray paths. Where separate imaging andillumination gratings are used the two gratings may employ different TIRangles within the waveguide. This is advantageous in terms of avoidingthe risk of cross coupling of illumination light into the detector andimage light into the light source.

In FIG. 1A the illumination light path is illustrated by the light 1010from the source which is directed into a TIR path 1011 by the inputprating and diffracted out of the waveguide as the light generallyindicated by 1012. Typically the eye tracker will have a pupil of size20-30 mm. Since the eye tracker will usually be implemented as part of aHMD its pupil should desirably match that of the HMD. FIG. 1 a showsreturn light 1013 reflected from the front surface of the cornea 117 andlight 1014 reflected from the retina 118. The corneal and retinal imagelight enters the waveguide along tray paths such 1015, 1116 and isdeflected into a TIR path such as 1017 by an active element of theimaging grating which is switching one element at a time. The light 1017is deflected into a ray path 1018 toward the detector by the outputgrating. The detector reads out the image signal in synchronism with theswitching of the SBG lens array elements. The detector is connected toan image processing apparatus for determining at least onespatio-temporal characteristic of an eye movement. The image processor,which is not illustrated, detects pre-defined features of thebackscattered signals from the cornea and retina. For example, the imageprocessor may be used to determine the centroid of an eye feature suchas the pupil. Other trackable features of the eye will be well known tothose skilled in arts of eye tracker design and visual optics.

The eye surfaces used for tracking are not necessarily limited to thefront surface of the cornea and the retina. The invention can be appliedusing reflections from any of the surfaces of the lens, iris and scleraincluding any of the reflections normally referred to as Purkinjereflections. In one particularly important embodiment of the inventionto be discussed later the light reflected from the eye is speckle. Thespeckle may arise from reflections at any of the above surfaces or fromthe bulk medium of the cornea lens and other parts of the eye.

Advantageously, the light source is a laser emitting in the infraredband. Typically, the laser emits at a wavelength in the range 785-950nm. The choice of wavelength will depend on laser efficiency, signal tonoise and eye safety considerations. Light Emitting Diodes (LEDs) mayalso be used. In one embodiment of the invention the detector is a twodimensional array. However other detector may be used including lineararrays and analogue devices such as position sensing detectors.

In the embodiment shown in FIG. 1 the illumination grating providesdivergent light. In alternative embodiments of the invention theillumination grating provides collimated light.

The gratings may be implemented as lamina within or adjacent an externalsurface of the waveguide. Advantageously the gratings are switchableBragg gratings (SBGs). In certain embodiments of the invention passivegratings may be used. However, passive gratings lack the advantage ofbeing able to direct illumination and collect image light from preciselydefined areas of the pupil. In one embodiment the gratings are reversemode SBGs. Although the invention is discussed in relation totransmission gratings it should be apparent to those skilled in the artthat equivalent embodiments using reflection gratings should be feasiblein most cases. The gratings may be surface relief gratings. However,such gratings will be inferior to Bragg gratings in terms of theiroptical efficiency and angular/wavelength selectivity.

The input and illumination gratings may be configured in many differentways. FIG. 2 is a schematic plan view showing one possibleimplementation for use with the embodiment of FIG. 1. Here input gratingcomprises two grating elements 114A,114B and the illumination grating isalso divided into the upper and lower gratings 120A,120B each providingnarrow beam deflecting grating strips above and below the imaginggrating 102. The detector grating 115 is also indicated. Since theguided beams in the input and illumination grating are collimated andlikewise the guided beams in the imaging and detector gratings there isno cross talk between the two regions of the waveguide.

In the embodiment of the invention shown in FIGS. 3-4 similar to the oneof FIG. 2 the upper and lower illumination grating may be arrays ofswitchable grating elements 121A,121B comprised switchable gratingelements such as 122A,122B. The SBG deflector arrays scroll illuminationacross the exit pupil in step with the activation of the imaging gratingelements. Finally in the embodiment of FIG. 4 the illumination gratingcomprises just one strip 123 containing elements 124 at the top edge ofthe imaging grating.

The invention does not assume any particular configuration of thegrating elements. It is important to note that the SBGs are formed ascontinuous lamina. Hence the illumination gratings elements may beconsidered to be part of the imaging grating. This is a significantadvantage in terms of fabrication and overall form factor. In embodimentwhere the illumination grating is split into two elements the inputlaser light may be provided by one laser with the upper and lower beambeing provided by a beam splitting means. Alternatively, two separatelaser modules may be used to provide light that is coupled into thewaveguide via the input gratings 114A,114B are illustrated in FIGS. 3-4.The invention does not assume any particular method for providing thelaser input illumination or coupling the laser light into the waveguide.Many alternative schemes should be apparent to those skilled in the artof optical design.

The illumination grating may provide illumination light of any beamgeometry. For examples the light may be a parallel beam emitted normallyto the surface of the eye tracker waveguide. The illuminator grating isillustrated in more detail in the schematic side elevation view of FIG.5. The SBG linear array 130 is sandwiched between transparent substrates130A,130B. Note that the substrate layers extended to cover the entirewaveguide and therefore also act as the substrates for the imaginggrating. Advantageously, the ITO layers are applied to the opposingsurfaces of the substrates with at least one ITO layer being patternedsuch that SBG elements may be switched selectively. The substrates andSBG array together form a light guide. Each SBG array element has aunique optical prescription designed such that input light incident in afirst direction is diffracted into output light propagating in a seconddirection. FIG. 5 shows TIR illumination beam 1020 being deflected bythe active element 131 to provide divergent illumination light 1021. Thegeometrical optics of has been simplified for the sake of simplifyingthe description.

An alternative embodiment of the linear deflector array is shown in theschematic side elevation view of FIG. 6. In this cases the array 132sandwich by substrates 132A,132B is based on a lossy grating thatdiffracts incrementally increasing fractions of the guided beam out ofthe waveguide towards the eye. Beam portions 1023A-1023C diffracted bythe grating elements 133A-133C are illustrated. Typically the indexmodulation of the grating elements will be designed to provide uniformextraction along the array and hence uniform output illumination.

Advantageously, the illumination grating elements encode optical powerto provide sufficient beam spread to fill the exit pupil with light. Asimilar effect may be produce by encode diffusion characteristics intothe gratings. The apparatus may further comprise an array of passiveholographic beam-shaping diffusers applied to the substrate overlaps thelinear SBG arrays to enhance the diffusion. Methods for encoding beamdeflection and diffusion into diffractive devices are well known tothose skilled in the art of diffractive optics. Cross talk between theimaging and illumination channels is overcome by configuring the SBGssuch that the illumination TIR path within the eye tracker lies outsidethe imaging TIR path.

In one embodiment of the invention the imaging grating may also encodeoptical power. A two layer SBG imaging grating that encodes opticalpower is illustrated in FIGS. 7-10. The arrays are shown in theirstacked configuration in FIG. 7. The substrates 136A,136B and 139A,139Btogether provide the imaging waveguide as illustrated in FIG. 8 wherethe ray path from the eye into the waveguide via an activated SBGelement 42 is represented by rays 1025-1028. The arrays are shown infront, plan and side elevation views in FIGS. 9-10. The arrays compriselinear arrays of column elements each having the optical characteristicsof a cylindrical lens. The column vectors in the two arrays areorthogonal. The first array comprises the SBG array 135 sandwiched bythe substrates 136A,136B with one particular element 137 beingindicated. The second array comprises the SBG array 40 sandwiched by thesubstrates 139A,139B with one particular element 141 being indicated.

FIG. 11A illustrates the principles of the formation of the first fourPurkinje images corresponding to reflections off the front of the cornea1033,1043; the back of the cornea 1032, 1042; the front of the eye lens1031,1041; and the back of the eye lens 1030,1040. FIG. 11B illustratesthe formation of images of the retina 1034,1044 and the iris 1035,1045.

FIG. 12 shows how the first and second SBG lens arrays of FIGS. 7-10 maybe used to localize an eye feature such as speckle by scanning row andcolumn SBG elements such as 142 and 143.

FIG. 13 illustrates how the size of speckle feature as recorded in twocaptured speckle images may vary with the eye orientation anddisplacement with respect to the eye optical axis 1050. FIG. 13Aillustrates speckle formed by illuminating the eye along the direction1050A which is initially parallel to the eye optical axis. Thecomponents of the corneal and retinal speckle light parallel to the eyeoptical axis are indicated by 1050B,1050C. FIG. 14A shows the formationof speckle with the eye rotated in the plane of the drawing. Thedetected corneal and retinal speckle light 1050D,1050E parallel to thedirection 1050 which is now no longer parallel to the eye optical axisis shown. As shown by the insets 1051,1053 the size and spatialdistribution of the speckles changes as the eye rotates. Correlation ofthe two speckle patterns will provide a measure of the eye rotation.Note that, typically, the speckle patterns recorded at the detector willcombine separate speckle patterns from the cornea and retina as well asother surfaces and biological media interacting with the illuminationbeam.

In one embodiment of the invention the eye tracker processor comparesthe speckle images due to light being scattered from the retina andcornea. When the eye is panned horizontally or vertically the relativeposition of the speckle pattern from the cornea and retina changeaccordingly allowing the direction of gaze to be determined from therelative trajectories of the reflected light beams. FIG. 14 representsthe front of the eye 146 cornea 147 and illuminated region 148 of theretina illustrates the direction of movement of corneal and retinalspeckle features as indicated by the vectors 149,150 correspond to theocular displaces illustrated in FIG. 15. FIG. 15A represents thereflection of rays from the cornea 1056,1057 and retina 1054,1055 forone eye position. FIG. 15B shows the reflection paths from the cornea1058,1059 and the retina 1060,1061 after a horizontal (or vertical) eyerotation.

In one embodiment of the invention based on the one of FIGS. 7-10 theimaging grating comprises an SBG array 143 in which the lens elements144 have varying focal length across the exit pupil. In the embodimentof FIG. 16 grating elements of first and second focal length indicatedby the divergent beams 1062,1064 and 1063,1065 are uniformlyinterspersed. In one embodiment of the invention illustrated in FIG. 17Athe imaging waveguide comprises arrays 145 of variable power lenselements 146. As shown in the detail of FIG. 17B a variable power lenswould be provided by combining a diffractive element 147 of fixed focallength with a variable index layer 148.

In one embodiment of the invention shown in the schematic view of FIG.18 the imaging grating comprises a single layer two dimensional SBGarray. A group of elements labelled 152 which comprises interspersedelements such as 153,154. The group forms the image region 151 at thedetector 110. Each SBG element is characterised by one from a set of atleast two different prescriptions. FIG. 18 does not show the details ofthe waveguide and the illumination and input/output gratings. At leastone of the SBG prescriptions corresponds to a lens for forming an imageof the eye on the detector. At least one prescription is optimised forimaging a speckle pattern formed by a surface of the eye. Hence theembodiment of FIG. 18 allows eye tracking to be performed using specklepatterns and conventional features such as Purkinje reflections.

An Embodiment Using Separate Illumination and Detection Gratings

FIGS. 19-24 provide schematic illustrations of aspects of an eye trackerbased on the principles of FIGS. 1-6. In this embodiment of theinvention the earlier described imaging, illumination, input and outputgratings are augmented by an additional grating to be referred to as animage sampling grating which overlays the output grating. FIG. 19 showsa side elevation view of the illumination grating 163. FIG. 20 is a planview showing the imaging grating 165, the illumination grating 163 andthe image sampling grating 170 overlaid on the output grating 164. FIG.21 is a side elevation view of an alternative embodiment of theillumination grating 163. FIG. 22A is a plan view of the imaginggrating, the image sampling grating 14 and the detector module 180. FIG.22B is a plan view of the image sampling grating and the detectormodule. FIG. 22C is a cross sectional view showing the imaging gratingand the image sampling grating. FIG. 22D is a cross sectional view ofthe image sampling grating and the detector module. Finally, FIG. 22E isa cross sectional view of the imaging grating, the image samplinggrating and the detector module. To assist the reader the projectionplane of each illustration is referred to a Cartesian XYZ referenceframe.

The imaging grating 165 comprises an array of column-shaped SBGelements, such as the one labelled 167, sandwiched by substrates168,169. Column elements of the imaging grating 165 are switched on andoff in scrolling fashion backwards and forward along the directionindicated by the block arrow 1320 in FIG. 20 such that only one SBGcolumn is in its diffractive state at any time.

The illuminator array 163 is shown in detail in FIG. 19 comprisessubstrates 161A,161B sandwiching an array of SBG rectangular elementssuch as 163A,163B. The SBG elements may have identical diffractingcharacteristics or, as shown in FIG. 19, may have characteristics thatvary with position along the array. For example, the element 163Aprovides a diffusion distribution 1310 centred on a vector at ninetydegrees to the array containing rays such as 1311. However, the element63B provides an angled distribution 1312 containing rays such as 1313.In an alternative embodiment shown in FIG. 21 the diffusion polardistributions may have central ray directions that varying in a cyclicfashion across the array as indicated by the rays 1313-1318.

The image sampling grating 170, comprising an array of rectangular SBGbeam deflecting elements 173 such as 176 (shown in its diffracting statein FIG. 22C) sandwiched by substrates 174,175. The waveguide containingthe imaging grating 165, illumination grating 163 and the output grating164 is separated from the image sampling grating 170 by a medium (notillustrated) which may be air or a low refractive index transparentmaterial such as a nanoporous material.

Infrared light from a surface of the eye is coupled into the waveguideby an active imaging grating element, that is, by a diffracting SBGcolumn. The guided beam undergoes TIR in the waveguide up to the outputgrating. As shown in FIG. 22C the output grating 164 deflects the beamthrough ninety degrees into the direction 1322 towards the imagesampling grating 170. As shown in FIG. 22C a portion of the beam 1322 isdeflected into the image sampling grating by an active SBG element 176where it undergoes TIR in the direction indicated by the ray 1323 (andalso by block arrow 1321 in FIG. 20). The light that is not sampled bythe image sampling grating indicated by 1320 1321 is trapped by asuitable absorbing material, which is not illustrated. The TIR beam isdeflected in the detector module 180 by a first holographic lens 172 toprovide out image light 1325. Turning now to FIG. 22D we see that thedetector module contains mirror surfaces 177A,177B and a furtherholographic lens 178 which forms an image of the eye features or specklepattern that is being tracked on the detector array 166. Note theholographic lens 172,178 may be replaced by equivalent diffractiveelements based on Bragg or surfaces relief gratings. Conventionalrefractive lens elements may also be used where size constraints permit.

FIG. 23 provides a system block diagram of the eye tracker of FIGS.19-22. The system modules comprise the imaging grating 300, illuminationgrating 301, illumination grating driver 302, illumination samplinggrating 303, imaging grating driver 304, detector driver 30, imagesampling array driver 306, detector 166 and processor 307. The apparatuswill also comprises a laser driver which is not illustrated. The opticallinks from the image grating to the image sampling array and the imagesampling array to the detector are indicated by the block arrows329,330. The processor 307 comprises a frame store 308 or other imagememory device for the storage of captured eye image or speckle patternframes and an image processor 309 further comprising hardware orsoftware modules for noise subtraction 310 and image analysis 311. Theprocessor further comprises hardware control module 312 for controllingthe illumination, imaging and image sampling grating drivers, all saidmodules operating under the control of a main processor 313. Data andcontrol links between components of the system are indicated by 319-325.In particular, each driver module contains switching circuitryschematically indicated by 326-328 for switching the SBG elements in theimaging grating, illumination grating array, and image sampling grating.

FIG. 24 illustrates the switching scheme used in the imaging grating andimage sampling grating. The illumination grating elements are switchedin phase with the imaging grating columns. Column element 165A of theimaging grating array 165 and element 170A of the readout array 170 arein their diffracting states. The projection (indicated by 170B) ofelement 170A on the column 65A defines an active detection aperture.Using such as scheme it is possible to track features of the eye using aX,Y localisation algorithm aided by predictions obtained from analysisof displacement vectors determined from successive frames.

An Embodiment Using a Common Illumination and Detection Grating

FIGS. 25-27 provide schematic illustrations of aspects of an eye trackerthat extends the embodiment of FIGS. 19-24 by introducing a furthergrating component to be referred to as an illumination sampling gratingwhich overlays the input grating. The other feature of this embodimentis that the illumination grating is no longer separate from the imaginggratings. Instead the two are combined such that a common column gratingarea is used to illuminate and image the eye with the illumination andimage wave guided light propagating in opposing directions. The combinedgratings will be referred to as the illumination and imaging grating. Aswill be explained below the function of the illumination samplinggrating, which is similar in structure to the image sampling grating, isto concentrate the available illumination into region of the eyeselected by the image sampling grating. This provides the dual benefitsof light efficiency and avoidance of stray light from regions of the eyethat are not being tracked. Turning to the drawings FIG. 25 is a planview showing the imaging and illumination grating 190, the imagesampling grating 194, illumination sampling grating 195 the inputgrating 193 and output grating 192 and the detector module 200. Columnelements of the illumination and imaging grating are switched on and offin scrolling fashion backwards and forward such that only one SBG columnis in its diffractive state at any time. The counter propagating beampaths are indicated by 1341,1342.

FIG. 26 shows the components of FIG. 25 in a side elevation view.

FIG. 27A is a plan view of the illumination sampling grating

FIG. 27B is a cross sectional view of the illumination sampling grating195 including the input grating 193 and the laser 205.

FIG. 27C is a plan view of the image sampling grating 194 showing thedetector module 200 and detector 205 overlaid.

FIG. 27D is a side elevation view showing detector module 200 in moredetail. The detector 205 and a cross section of the image samplinggrating 194 are included.

FIG. 27E is a cross sectional view of the output grating 192 and theimage sampling grating 194.

FIG. 27F is a cross section view of the input grating 193 and theillumination sampling grating 194.

To assist the reader the projection plane of each illustration isreferred to a Cartesian XYZ reference frame.

The illumination and imaging grating comprises the array 190 ofcolumn-shaped SBG elements, such as the one labelled 191 sandwiched bythe transparent substrates 190A,190B. The input and output grating whichare disposed in the same layer are labelled by 193,192 respectively. Thedetector module 200 is delineated by a dotted line in FIGS. 25-26 and inmore detail in FIG. 27D.

The image sampling grating 194, comprises an array of rectangular SBGbeam deflecting elements (such as 197) sandwiched by substrates194A,194B. Typically the imaging grating and image sampling grating areseparated by a medium 198 which may be air or a low refractive indextransparent material such as a nanoporous material.

The illumination sampling grating 195 which is has a very similararchitecture to the image sampling grating comprises an array ofrectangular SBG beam deflecting elements (such as 196) sandwiched bysubstrates 195A,195B. Typically the imaging grating and image samplinggrating are separated by a medium 199 which may be air or a lowrefractive index transparent material such as a nanoporous material.

Referring to FIG. 26 and FIG. 27F illumination light 1350 from the laseris directed into the illumination sampling grating by a coupling grating207. The light then proceeds along a TIR path as indicated by 1350A,1350B up to an active element 208 where it is diffracted into thedirection 1351 towards the input grating. Not that the image samplinggrating directs all of the illumination light through the active elementof the illumination sampling grating the elements of which are switchedin synchronism with the elements of the image sampling grating to ensurethat at any time the only the region of the that is being imagedreceives illumination. The illumination path in the waveguide isindicated by 1352-1354.

Infrared light 1356 (also illustrated as the speckle pattern 1355) froma surface of the eye is coupled into the waveguide by a diffracting SBGcolumn such as 191. The guided beam indicated by 1357,1358 undergoes TIRin the waveguide up to the output grating 192. The output gratingdeflects the beam through ninety degree into the direction 1359 towardsthe image sampling grating. As shown in FIG. 27E the beam in direction1359 is deflected into the image sampling grating by an active SBGelement 197 where it undergoes TIR along the ray path indicated by 1360,1361. The TIR beam is deflected into the detector module 200 as light1363 by a first holographic lens 203. Any light that is not sampled bythe image sampling grating is trapped by a suitable absorbing material,which is not illustrated.

The detector module contains mirror surfaces 201,202 and a furtherholographic lens 204 which forms an image of the eye features or specklepattern that is being tracked on the detector array 205. The ray pathfrom the image sampling grating to the detector is indicated by the rays1363-1365. Note the holographic lens 203,204 may be replaced byequivalent diffractive elements based on Bragg or surfaces reliefgratings. Conventional refractive lens elements may also be used wheresize constraints permit.

In one embodiment of the invention illumination light from laser moduleis converted into S polarized light which is coupled into the eyetracker waveguide by the input grating. This light is then convertedinto circularly polarized light using a quarter wave plate. An activeSBG column will then diffract the P-component of the circularlypolarized wave guided light towards the eye, the remaining P-polarizedlight being collected in a light trap. The P-polarized light reflectedback from the eye (which will be substantially P-polarized) is thendiffracted into a return TIR path by the active SBG column and proceedsto the detector module as described above. This scheme ensures thatimage and illumination light is not inadvertently coupled into the inputand output gratings respectively. In other embodiments of the inventionthe unwanted coupling of the image and illumination light may beovercome by optimizing the TIR angles, the angular bandwidths of theimaging and illumination gratings, the spacings along the waveguide ofthe input and output gratings, and the illumination and imaging beamcross sections. In one embodiment the illumination light which willtypically in most embodiments of the invention be collimated may beangled such that the waveguide propagation angle of the illuminationbeam differs from the waveguide angles of the image light.

An important feature of the invention is that elements of theillumination sampling grating are switched to allow illumination to belocalized to a small region within the active column of the DigiLensensuring that the illumination is concentrated exactly where it isneeded. This also avoids stray light reflections a problem which canconsume significant image processing resources in conventional eyetracker designs.

Since the illumination is scrolled the cornea and retina are not exposedto continuous IR exposure allowing higher exposures levels to be usedleading to higher SNR. A safety interlock which is not illustrated maybe included to switch off the laser when no tracking activity has beendetected for a predefined time.

The proposed scheme for switching the columns and readout elements inthe embodiments of FIGS. 25-27 is based on tracking the movement of thepupil using a X,Y localisation algorithm similar to the one illustratedin FIG. 24 which shows the how the ith activation column DigiLens andjth activated element of the Readout Array are used to select thespeckle pattern region (X,Y).

The detected speckle pattern (or other eye signature) is stored andcompared with other saved patterns to determine the eye gaze trajectoryand to make absolute determinations of the gaze direction (boresighting). Initial calibration (that is, building up the database ofsaved patterns) is carried out by directing the user to look at testtargets at predefined points in the FOV. As discussed above the eyetracker tracks eye movements by measuring the spatio-dynamiccharacteristics of speckle patterns projected off the cornea and retina.Speckle detection avoids the image analysis problems of identifying andtracking recognisable features of the eye that are encountered inPurkinje imaging schemes. Instead we detect and correlate specklepatterns (or other eye signatures) using a spatio-temporal statisticalanalysis. A prerequisite is achieving an adequate level of specklecontrast after detector noise and ambient light have been subtractedfrom the detected signal and being able to resolve speckle grains.

FIG. 28 is a simplified representation of the detection path startingwith the collimated rays 1400 from an active column element 370 of theimaging array. The rays 1400 are sampled by an element 371 of thedetector grating to provide the rays 1402 which are imaged by theholographic lens 372 to provide the rays 1403 incident on the detector205.

FIG. 29 provides a system block diagram of the eye tracker of FIGS.26-27. The system modules comprise the illumination and imaging grating190, image sampling grating 194, illumination sampling grating 195,detector 205, laser 206, illumination sampling array driver 340, imagesampling array driver 341, detector driver 342, laser driver 343,illumination and imaging grating driver 344 and processor 345. Theprocessor 345 comprises a frame store or other image storage media 346for the storage of captured eye image or speckle pattern frames and animage processor 347 further comprising hardware or software modules fornoise subtraction 348 and image analysis 349. The processor furthercomprises hardware control module 350 for controlling the illumination,imaging and image sampling grating drivers, all said modules operatingunder the control of a main processor 351. In one embodiment of theinvention the detector array is a detector array of resolution 16×16with a framing rate of 2300 fps of the type commonly used in infraredmouse equipment. The above described modules are connected bycommunication and control links schematically indicated by 360-369include control lines for switching the SBG elements in the imaginggrating, illumination sampling grating array, and image sampling grating367-369.

Prerequisites for measuring eye displacement vectors (rotational and/ortranslational) include achieving an adequate level of speckle contrast(after detector noise and ambient light have been subtracted from thedetected signal) and being able to resolve individual speckle grains. Ahigh signal to noise ratio (SNR) is essential for detecting variationsin speckle properties at required angular resolution. The SNR depends onthe speckle contrast, which is defined as the ratio of the root meanssquare (rms) variation of the speckle intensity to the mean intensity.The speckle contrast lies between 0-1 assuming Gaussian statistics. Thedetector should have low noise and a short integration time. If themotion of the eye is appreciably faster than the exposure time of theCCD camera rapid intensity fluctuations of the speckle pattern willoccur, the average of the detected patterns resulting in a blurred imagewith reduced speckle contrast.

The smallest speckle size is set by the diffraction limit. Applying thewell known formula from diffraction theory: w=˜2.44 D/a (assuming: adetector lens to detector distance D˜70 mm.; IR wavelength λ=785 nm.;and detector lens aperture a˜3 mm.) we obtain a diffraction limitedspeckle diameter w at the detector of ˜64 microns. The resolution of atypical mouse sensor is around 400-800 counts per inch (cpi), with ratesof motion up to 14 inches per second (fps). Hence the limiting specklesize is equivalent to one count per 64 micron at 400 cpi which isroughly compatible with the expected speckle size.

Ideally the eye tracker should be capable of tracking the eye's gazedirection everywhere within the eye box and for the full range of eyerotations. For the most demanding applications the design goal is toresolve 0.15° over the entire FOV. In the case of speckle trackers it isimportant to emphasize that we are not tracking ocular features in theconventional way. Instead we are measuring eye displacement vectors bycomparing speckle patterns using statistical correlation techniques. Asthe eye translates and rotates within the eye box the DigiLens columnsand readout elements select X-Y addressed speckle patterns (includingcorneal and retinal components) which are sequentially imaged onto adetector.

The processes of tracking and bore sighting are aided by recording largenumbers of reference speckle pattern frames for different eye positionsand orientations. Since the frames are of low resolution large numbersof samples may be collected without significant computational overhead.

The detector optical prescription will be determined by a detailedray-tracing analysis and will require trade-offs of speckle size,F-number and DigiLens column width. The detector lens aperture definesthe limiting speckle size. The detector field of view is determined bythe detector size and the detector lens focal length. At present thepreferred detector is the Agilent IR Mouse Sensor which uses a 16×16element photo detector array.

In one embodiment the DigiLens provides 25 SBG scrolling columns×17 SBGreadout elements. The Agilent device can be programmed to switch 2300fps So a complete scan of the FOV will take (25×17)/2300 s.=185 ms.However, in practice the eye tracker will use a more sophisticated X-Ysearch process that localises the pupil using column and readout elementcoordinates. It is anticipated that on average around 10 search stepsmay be needed to converge on the pupil position resulting in a latencyof 4.3 ms. On this basis the latency of the tracker is potentially ×100lower than that of comparable image processing-based Purkinje-type eyetrackers. It is also anticipated that the correlation process will beimplemented in hardware resulting in a relatively modest data processinglatency.

The proposed strategy for processing speckle data captured by the eyetracker is based on the following assumptions.

-   -   a) Speckle patterns provide unique “fingerprints” of regions of        the cornea and retina.    -   b) Unlike speckle interferometry which requires that the speckle        motion is less than speckle size, speckle imaging using a        detector array requires that the speckle displacement from frame        to frame is greater than the speckle size    -   c) A displacement of the cornea and retina relative to the        detector will result in a shift of the speckle pattern by the        same amount    -   d) The shifts of the corneal and retinal speckle patterns will        be in opposite directions.    -   e) The motion of the speckles can be determined from the        correlation of two consecutive frame speckle patterns. This        information together with the relative motion of the corneal and        retinal speckle patterns can be used to determine eye        displacement vectors.    -   f) The correlation and image analysis processes may take        advantage standard techniques already developed in applications        such as radar, biological imaging etc.    -   g) The speckle contrast and speckle size at the detector are        compatible with the detector resolution and SNR.    -   h) An IR mouse detector such as the Agilent ADNS-2051 16×16        detector will be suitable.

The flow chart in FIG. 30 summarizes the process for determining eyedisplacement vectors from the recorded speckle data. The process relieson a database of frame data collected during initial calibration andnoise characteristics. The calculation of the displacement vectors usesinputs from a suite of mathematical models that simulate the first ordereye optics, the eye tracker optics and the eye dynamics. The process maybe interrupted by the user or automatically when a DigiLens failureoccurs. The process also includes DigiLens hardware control to enableX,Y addressing of DigiLens columns and readout elements.

The correlation process for obtaining the eye displacement vector fromtwo detected frames in one embodiment may be summarised as follows. Eachframe is subdivided into small sub frames. The sub-frame coordinates maybe predefined or alternatively may be determined by an interactivescheme using the output from an Eye Dynamics Model. A 2D correlation mapbetween the sub images from the two frames is calculated starting with aone pixel step in the x and y directions and repeat the calculationincreasing the step size by one pixel at a time. Other statisticalmetrics may also be computed at this stage to assist in refining thecalculation. We then repeat the correlation process for another selectedframe region. A displacement vector is then computed using (for the timeperiod between the two analysed frames) using the peaks of thecorrelation maps. Ideally the sub frames should be entirely within thecorneal or retinal fields, the two being distinguished by their opposingdirections. Data which does not yield clear separation of the two willbe rejected) at this stage. The calculation is refined using data froman Eye Optical Model which models of the eye dynamics and an Eye TrackerModel which models the optical system. The verified displacement vectoris used to determined the next search X,Y coordinates (ie SBG column,row) for the Eye Tracker using predicted gaze trajectory calculatedusing a Eye Dynamics Model. The basic ray optics used in the Eye Modelin particular the relationship of the first order corneal and retinalreflection paths of the eye may be modelled using ray-tracing programmessuch as ZEMAX. Standard eye models well known to those skilled in theart will be adequate for this purpose. Further models may be used tosimulate speckle from the retina and the cornea. The Eye Dynamics Modelcarries out a statistical analysis of the displacement vectors fromprevious frames to determine the most optical next X,Y search location(ie the columns and readout elements to be activated in the DigiLens.

Reflection from the cornea has a strong secular component. Retinalreflection is more diffuse. The size of the corneal reflected angleswould ordinarily require a large angular separation between theillumination and detection optical axes. This would make eye trackingusing corneal reflections over large FOVs very difficult. The inventionavoids the problem of imaging large reflection angles (and dealing withare lateral and vertical eye movements which can arise from slippage) byusing matched scrolling illumination and detection. Hence the reflectionangle becomes relatively small and can be approximated to:Ψ˜2[(D/r−1)Φ+d/r] where r is the cornea radius Φ is the eye rotation andD is the distance of the eye centre from the displaced centre ofcurvature of the cornea and d is the lateral displacement of the eyecentre.

Initial calibration is carried out by directing the user to look at testtargets at predefined points in the FOV. The bore-sighting process isillustrated in FIG. 32 which shows a flowchart (FIG. 32A) and aschematic illustrates of the initial calibration procedure (FIG. 32B).According to FIG. 31A the bore sighting procedure 400 comprises thefollowing steps:

At step 401 present targets to the eye at location j;

At step 402 capture a series of frames at location j;

At step 403 store the capture frames;

At step 404 move to the next target position in the field of view (FOV).

At step 405 repeat the process while j is less than a predefined integerN; otherwise end the process (at step 406).

Referring to FIG. 31B we see that initial calibration will be carried bypresenting targets (typically lights sources, resolution targets etc) tothe viewer at different points 1≦j≦N in the field of view 410 (the pointalso being labelled as 411-413) and capturing and storing frames ofspeckle pattern images at each location. The targets may be presentedsequentially along the sweep path labelled by 414. However, otherpresentation schemes may be used. The stored frames will be processed toenhance SNR and extract statistical metrics (such as histograms,probability density functions for speckle size etc) for subsequent“on-the-fly” frame comparison. Each frame provides a “fingerprint” forthe region of the FOV concerned. The signatures will vary in: relativepositions of the corneal and retinal speckle pattern; speckle contrast;and the speckle size distribution (linked to optical magnification).

The optical design requires careful balancing of the high source fluxneeded to overcome throughput inefficiencies arising from the smallcollection angles, low transmission thorough the DigiLens and the lowreflectivity of the eye (˜2.5% at the surface of the cornea) with therequirement for eye-safe IR illumination levels. Typically, forapplications in which the eye tracker is used for hours at a time undercontinuous IR exposure the eye irradiance should not exceed around 1mW/cm2. The appropriate sstandards for eye safe infrared irradiance arewell known to those skilled in the art. Since in our eye tracker wescroll the illumination the cornea and retina are not exposed tocontinuous IR exposure allowing higher exposures levels to be usedleading to higher speckle contrast level and therefore higher SNR at thedetector. In a SBG design there is the risk of a switching malfunctioncausing the laser beam scanning to freeze resulting in all of theavailable output laser power being concentrated into a small area of theeye. To overcome this problem a safety interlock will be provided toswitch off the laser when no tracking activity has been detected for apredefined time—typically a few minutes. During this dead time the IRexposure may be allowed to increase significantly without exceeding thesafety threshold, as indicate by the graph.

The following characteristics of the speckle image may also be used toassist the tracking of the eye use speckle: speckle grain size; specklebrightness (either individual or collective brightness); speckle shape;rate of change of any of the preceding characteristics with ocularmovement; and relative directions of corneal and retinal bemadisplacements. It is further recognized that each of these aspects ofthe speckle image will be dependent on the illumination beam direction(scanning or static); the detection optics and the focal length of theimaging optics. The rate of change of the corneal versus retinalspeckles will depend on the focal length.

As discussed the eye tracker measures and compare the signals from awide range of “scanned” horizontal and vertical positions in front ofthe eye with calibration images recorded for the subject eye. Using theabove speckle characteristics, the gaze direction may be determined toprogressively greater levels of resolution and accuracy according to thenumber of characteristics measured. Advantageously, the user wouldcalibrate the tracker by looking ahead and into the top/bottom left andright hand corners of the FOV. However, this may not be necessary in allembodiments of the invention. In addition by measuring the retinal andcorneal speckles patterns and using more than one characteristic it ispossible to determine the absolute gaze direction as well as therelative displacement.

Speckle tracking avoids the cost and complexity of implementingclassical Purkinje imaging methods. Conventional iris image capturesystems are an indicator the level of processing that will be requiredin an eye tracker. The iris image is typically acquired by a camerausing infrared light in the 700 nm-900 nm band resolving in the regionof 100-200 pixels along the iris diameter. The first step is usually todetect and remove stray light before proceeding to determine theboundaries of the iris. Typically the centers and radii of iris andpupil are approximated initially by applying a circular edge detector.High accuracy and rapid response times require high-performance andhigh-cost microprocessors that are beyond the scope of consumerproducts. Traditional image processing designs based on software are tooslow. It is known that significant improvements may result from an aniris recognition algorithms based on a hardware-software co-design usinglow-cost FPGAs The system architecture consists of a 32-bit generalpurpose microprocessor and several dedicated hardware units. Themicroprocessor executes in software the less computationally intensivetasks, whereas the coprocessors speed-up the functions that have highercomputational cost. Typically, depending on the function implemented,coprocessors speed-up the processing time by a factor greater than 10compared to its software execution. However, the best latency achievedwith hardware-software co-designs, is typically in the range 500-1000ms. It should be noted that an eye tracker is a much more demandingproposition for an image processor. Detecting a clean iris image is onlythe first step. Applying the edge detection algorithms as the eye movesaround the eye box will require several frames to be analysed adding tothe overall latency.

Table 1 presents a comparison of an eye tracker based on a single SBGlayer DigiLens as discussed above with a conventional image sensorcomprising a camera and image recognition algorithms in the table below.

TABLE 1 Comparison of the present invention and a camera/imageprocessing eye tracker. Camera and Speckle Eye Tracker Image ProcessingDetector 16×16 IR Mouse VGA CMOS Camera Sensor Frame Rate 2300 fps 60 HzDetector Latency 0.43 ms (for 16 × 16 16.67 ms (for VGA frame) pixelimage frame) Image Processing 4.3 ms. Estimated: 500 ms.-1000 ms.Latency (10 frames using (To apply feature X-Y search algorithm).recognition and tracking). Total Eye Tracker ~5 ms. ~500-1000 ms.Latency Relative Latency 1 VGA CMOS Camera

The proposed eye tracker is compatible with many display applications inconsumer products, avionics and other fields such as Augmented Realityby enabling the features of: wide field of view; large exit pupil; thinform factor; low inertia; and easy integration with near-eye displaytechnologies.

It should be emphasized that the drawings are exemplary and that thedimensions have been exaggerated. For example thicknesses of the SBGlayers have been greatly exaggerated.

In any of the above embodiments the substrates sandwiching the HPDLClayer may be planar, curved or formed from a mosaic of planar or curvedfacets.

An eye tracker based on any of the above-described embodiments may beimplemented using plastic substrates using the materials and processesdisclosed in PCT Application No.: PCT/GB2012/000680, entitledIMPROVEMENTS TO HOLOGRAPHIC POLYMER DISPERSED LIQUID CRYSTAL MATERIALSAND DEVICES.

Advantageously, the SBGs are recorded in a reverse mode HPDLC materialin which the diffracting state of SBG occurs when an electric field isapplied across the electrodes. An eye tracker based on any of theabove-described embodiments may be implemented using reverse modematerials and processes disclosed in PCT Application No.:PCT/GB2012/000680, entitled IMPROVEMENTS TO HOLOGRAPHIC POLYMERDISPERSED LIQUID CRYSTAL MATERIALS AND DEVICES.

However, the invention does not assume any particular type of SBG.

A glass waveguide in air will propagate light by total internalreflection if the internal incidence angle is greater than about 42degrees. Thus the invention may be implemented using transmission SBGsif the internal incidence angles are in the range of 42 to about 70degrees, in which case the light extracted from the light guide by thegratings will be predominantly p-polarized.

Using sufficiently thin substrates the eye tracker could be implementedas a long clear strip appliqué running from the nasal to ear ends of aHMD with a small illumination module continuing laser dies, light guidesand display drive chip tucked into the sidewall of the eyeglass. Astandard index matched glue would be used to fix the display to thesurfaces of the HMD.

The method of fabricating the SBG pixel elements and the ITO electrodesused in any of the above-described embodiments of the invention may bebased on the process disclosed in the PCT Application No. US2006/043938,entitled METHOD AND APPARATUS FOR PROVIDING A TRANSPARENT DISPLAY.

The invention does not rely on any particular methods for introducinglight from a laser source into the eye tracker and directing lightscattered from the eye onto a detector. In the preferred embodiments ofthe invention gratings are used to perform the above functions. Thegratings may be non switchable gratings. The gratings may be holographicoptical elements. The gratings may be switchable gratings.Alternatively, prismatic elements may be used. The invention does notrely on any particular method for coupling light into the display.

It should be understood by those skilled in the art that while thepresent invention has been described with reference to exemplaryembodiments, it is to be understood that the invention is not limited tothe disclosed exemplary embodiments. Various modifications,combinations, sub-combinations and alterations may occur depending ondesign requirements and other factors insofar as they are within thescope of the appended claims or the equivalents thereof.

What is claimed is:
 1. A eye tracker comprising: a waveguide forpropagating illumination light towards an eye and propagating imagelight reflected from at least one surface of an eye; a light sourceoptically coupled to said waveguide; a detector optically coupled tosaid waveguide; and disposed in said waveguide at least one gratinglamina for deflecting said illumination light towards said eye along afirst waveguide path and deflecting said image light towards saiddetector along a second waveguide path.
 2. The apparatus of claim 1wherein at least one portion of said at least one grating laminadeflects said illumination light out of said first waveguide path and atleast one portion of said at least one grating lamina deflects saidimage light into said second waveguide path.
 3. The apparatus of claim 2wherein said grating lamina comprises a multiplicity of electricallyswitchable elements each having a a diffracting state and a nondiffracting state, wherein each said at least one portion of said atleast one grating lamina is a grating element in its diffracting state.4. The apparatus of claim 1 wherein said first and second waveguidepaths are in opposing directions.
 5. The apparatus of claim 3 whereinsaid electrically switchable elements are elongate with longer dimensionaligned perpendicular to at least one of said first and second waveguidepaths.
 6. The apparatus of claim 1 wherein said at least one gratinglamina comprises an illumination grating for deflecting saidillumination light in said first waveguide path towards said eye; and animaging grating for deflecting said image light into said secondwaveguide path.
 7. The apparatus of claim 1 wherein said at least onegrating lamina further comprises at least one of: an input grating fordeflecting illumination light from said source into said first waveguidepath; and an output grating for deflecting said image light out of saidsecond waveguide path towards said detector.
 8. The apparatus of claim 7further comprising a image sampling grating overlaying said outputgrating, said image sampling grating comprising a linear array ofswitchable grating elements, each said element when in its diffractingstate sampling a portion of said light in said waveguide and deflectingit along said image sampling grating towards said detector.
 9. Theapparatus of claim 7 further comprising an illumination sampling gratingoverlaying said input grating, said illumination sampling gratingoptically coupled to said light source, said illumination samplinggrating comprising a linear array of switchable grating elements, eachsaid element when in its diffracting state deflecting light from saidillumination sampling grating into said waveguide.
 10. The apparatus ofclaim 6 wherein said illumination grating abuts an upper or lower edgeof said imaging grating along said first waveguide path.
 11. Theapparatus of claim 6 wherein said illumination grating comprises firstand second gratings disposed adjacent upper and lower edges of saidimaging grating along said first waveguide path.
 12. The apparatus ofclaim 6 wherein said imaging grating comprises a first array ofswitchable elongate beam deflection grating elements and an overlappingsecond array of switchable elongate beam deflection grating elements,said elements of said first and second arrays being disposed with theirlonger dimensions orthogonal.
 13. The apparatus of claim 6 wherein saidillumination grating is a linear array of elongate switchable beamdeflection elements with longer dimension aligned perpendicular to saidfirst waveguide path.
 14. The apparatus of claim 1 wherein said at leastone grating lamina is one of a switchable Bragg grating, a switchablegrating recorded in a reverse mode holographic polymer dispersed liquidcrystal, and a non switching Bragg grating.
 15. The apparatus of claim 1wherein said image light is speckle.
 16. The apparatus of claim 1wherein said eye surface is at least one of the cornea, lens, iris,sclera and retina.
 17. The apparatus of claim 1 wherein said detector isa two dimensional array.
 18. The apparatus of claim 1 wherein said atleast one grating lamina encodes optical power.
 19. The apparatus ofclaim 1 wherein said detector is connected to an image processingapparatus for determining at least one spatio-temporal characteristic ofan eye movement.
 20. The apparatus of claim 1 wherein said at least onegrating lamina encodes diffusing properties.